Variational inference for Student-t MLP models

This paper presents a novel methodology to infer parameters of probabilistic models whose output noise is a Student-t distribution. The method is an extension of earlier work for models that are linear in parameters to nonlinear multi-layer perceptrons (MLPs). We used an EM algorithm combined with variational approximation, the evidence procedure, and an optimisation algorithm. The technique was tested on two regression applications. The …rst one is a synthetic dataset and the second is gas forward contract prices data from the UK energy market. The results showed that forecasting accuracy is signi…cantly improved by using Student-t noise models. Key words: Variational inference, Student-t noise, multilayer perceptrons, EM algorithm, forecast. 1. Introduction In forecasting models, we generally assume that the data is corrupted by noise: yt = f(xt) + "t, where "t is a zero-mean probability distribution. Normally, the noise is assumed to be Gaussian distribution either because of arguments derived from the central Corresponding authour: email address: [email protected], Fax: +44 121 204 3685, Tel: +44 121 257 7718 Preprint submitted to Elsevier July 30, 2010 limit theorem or just to simplify calculations. For example, the log likelihood of a Gaussian noise model is a quadratic function of the output variables. This leads to the fact that in the training process, we can easily estimate the maximum likelihood solution using optimisation algorithms. Software and frameworks for training machine learning models such as radial basis functions (RBF), MLP, and linear regression (LR) with Gaussian noise can be found in [1]. Conversely, other noise models are much less tractable. So why use the Student-t distribution? In our previous work [2, 3], we used models with Gaussian noise to forecast gas and electricity forward prices in the UK energy market. In these experiments, the kurtosis, which is a measure of how outlier-prone a distribution is, of the residuals (i.e. the di¤erent between target and output of forecasting model) is between 16 and 17: the kurtosis of the Gaussian distribution is 3. Furthermore, P ( 3 < r < + 3 ) 0:982, where and are the mean and standard derivation of the residual respectively. The equivalent probability for a Gaussian distribution is 0:997; therefore, the residual distribution has heavy tails. This means that the residual distributions are much more outlier-prone than the Normal distribution. The large number of outliers can make the training process unreliable and error bar estimates inaccurate, because Gaussians are sensitive to outliers. It is clear that this data is not modelled well by a Gaussian distribution as has often been noted for …nancial data. As a consequence, a Student-t distribution can be considered as a good alternative to a Gaussian because it is a fat-tailed distribution and is more robust. Moreover, the Student-t distribution family contains the Normal distribution as special case. There are several previous studies of inference with Student-t models. Tipping and Lawrence proposed a framework for training an RBF model with …xed basis functions [4]. This study is a fully Bayesian treatment based on a varia-